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1  Introduction 


This  document  describes  the  usability  test  conducted  by  the  Air  Force  Research 
Laboratory  and  the  University  of  Dayton  Research  Institute  to  assess  user  performance  with  the 
Point  of  Maintenance  (POMX)  system  for  maintenance  data  collection  on  the  flightline.  This 
test  consisted  of  observations  of  task  performance  in  baseline  Corps  Automated  Maintenance 
Systems  (CAMS)  and  batch  POMX  conditions,  and  feedback  questionnaires  about  the  usability 
of  both  the  baseline  and  test  devices.  The  purpose  of  the  evaluation  was  to  compare  the  POMX 
device  with  the  current  CAMS  system  for  maintenance  documentation  (i.e.,  opening  work 
orders,  ordering  parts,  and  closing  work  orders).  The  POMX  device  used  was  the  Intermec  710, 
and  the  CAMS  system  consisted  of  a  standard  desktop  computer  using  CAMS  “green  screens.” 
Both  systems  were  connected  to  live  databases.  The  POMX  system  was  designed  to  allow  for 
both  a  live  mode  whereby  transmissions  could  be  made  via  radio  frequency  (RF),  and  a  batch 
mode  whereby  the  hand-held  device  needed  to  be  placed  into  a  cradle  for  interaction  with  the 
POMX  server.  In  both  of  these  modes,  however,  the  design  provided  for  portions  of  the  POMX 
database  to  be  loaded  onto  the  device  prior  to  its  use.  The  design  allowed  for  U.S,  Air  Force 
personnel  to  use  the  devices  in  a  flightline-type  setting  to  simulate  opening  work  orders,  ordering 
parts  and  closing  work  orders. 

This  test  was  designed  to  address  several  functions  from  the  Functional  Requirements 
Document  (Point  of  Maintenance  (POMX)  FY01,  Functional  Requirements  Document, 

Revision:  lb,  May  8, 2001).  These  functional  requirements  include:  33.3/2,  which  requires  the 
POMX  device  to  operate  in  both  a  standalone  mode  and  via  a  RF  or  LAN,  and  3. 3.7/1,  which 
requires  the  POMX  device  to  have  an  adequate  display  size. 

While  the  design  of  POMX  and  this  evaluation  were  intended  to  handle  opening  jobs, 
ordering  parts,  and  closing  work  orders,  at  the  time  of  the  evaluation  the  first  year  software  and 
hardware  configuration  was  unable  to  handle  ordering  parts  and  closing  work  orders  in  either  the 
batch  or  live  modes.  During  the  first  day  of  testing,  the  evaluation  team  and  the  Hurlburt  system 
administrators  were  unable  to  get  the  database  to  download  onto  the  POMX  device.  On  the 
second  day,  two  of  the  devices  loaded  the  necessary  POMX  database  files  onto  the  device; 
however,  other  problems  emerged.  Problems  with  RF  connectivity  prohibited  the  use  of  POMX 
devices  in  the  live  mode.  Additionally,  there  were  problems  with  first  year  software  designs  for 
drop  down  menus  in  the  forms  for  parts  ordering  and  closing  work  orders.  Some  of  the  drop 
down  menus  took  between  several  to  20  minutes  to  actuate.  The  evaluation  team  determined  that 
this  software  problem  was  unacceptable,  and  re-scoped  this  study  to  include  a  limited  number  of 
conditions  (based  on  those  functions  that  were  deemed  to  be  working  properly). 

Given  the  problems  discussed  above,  the  conditions  tested  included  CAMS  open  work 
orders,  CAMS  order  parts,  CAMS  close  work  orders,  and  POMX  open  work  orders.  This  report 
will  focus  only  on  the  differences  and  similarities  between  the  CAMS  open  work  order  and 
POMX  batch  mode  open  work  order  conditions.  The  intent  is  to  return  at  a  later  date,  after  Year 
2  POMX  software  is  installed  and  evaluate  the  POMX  live  open  work  order,  order  parts,  and 
close  work  order,  and  batch  order  parts,  and  close  work  order  conditions.  These  POMX 
conditions  will  be  compared  to  the  CAMS  conditions  tested  during  the  current  evaluation. 

During  the  testing  period,  other  observations  were  made  concerning  the  usefulness  and 
usability  of  the  Year  1,  existing  POMX  system.  These  observations  are  summarized  in  the 
Summary  and  Key  Findings  section  of  this  report. 
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1.1  Objectives 

This  usability  test  was  designed  to  address  the  following  usability  issues: 

•  For  opening  jobs,  ordering  parts  and  closing  jobs  on  the  flightline,  how  does 
mode  of  operation  (real-time  with  POMX,  batch  with  POMX  or  baseline  without 
POMX)  affect  performance?  NOTE:  due  to  system  and  software  problems 
the  only  POMX  objective  met  by  the  current  test  was  for  opening  jobs  on  the 
flightline  in  a  batch  mode. 

•  Which  mode  of  operation  do  users  prefer?  NOTE:  due  to  system  and  software 
problems  the  only  preference  information  gathered  was  relative  to  the 
POMX  batch  mode  for  opening  work  orders  in  comparison  with  opening 
work  orders  with  CAMS. 

•  Is  information  presented  on  POMX  device  readable  and  usable? 

1.2  Test  Methodology 

Usability  test  methods  applied  in  the  current  evaluation  are  based  on  principles  outlined 
by  Dumas  and  Redish  (1993).  In  accordance  with  these  usability  testing  methods,  design  of  the 
study  included  three  steps  (see  Figure  1).  First,  the  major  usability  issues  were  identified.  These 
included  concerns  about  the  general  hardware  use  (e.g.,  the  ability  of  the  device  to  operate  in  a 
standalone  or  connected  mode).  From  these  general  concerns,  specific  concerns  were  identified 
(see  the  objectives  above).  Finally,  methods  for  collecting  information  relevant  to  these 
concerns  were  identified.  Methods  included  multiple  metrics  for  each  concern  (e.g.,  collecting 
time  measurements  for  completion  of  specific  trials,  and  collecting  subjective  information). 

Using  this  method,  once  the  data  is  collected,  data  analysis  leverages  the  strength  of 
triangulating  the  information  gathered  for  each  specific  concern.  For  example,  the  specific 
concern  dealing  with  how  mode  affects  performance  included  use  of  three  metrics:  subject 
ratings,  experimenter  observations  of  the  participant  while  using  the  device,  and  specific 
participant  comments  (both  written  and  verbalized,  during  the  test  and  in  post-test  discussions). 
Using  triangulation,  each  metric  is  used  to  confirm  findings  of  another  metric  so  that  if  all  three 
metrics  identify  the  same  issue,  the  strength  of  usability  concern  is  increased.  Similarly,  as 
multiple  participants  identify  the  same  usability  problem,  the  usability  problem  is  revealed. 
Methods  used  to  analyze  results  are  fully  addressed  in  the  Results  section  of  this  document. 

In  identifying  usability  concerns,  research  indicates  that  the  number  of  participants  need 
not  be  as  high  as  tests  aimed  at  determining  significance  (e.g.,  ANOVAs,  or  tests  of  correlation). 
Virzi  (1992)  identifies  that  4  to  5  subjects  identify  80%  of  the  usability  problems  with  a  system, 
and  that  additional  subjects  are  less  likely  to  identify  any  new  problems.  The  current  test  was  not 
only  aimed  at  looking  at  the  usability  of  the  system  but  on  statistical  comparison  of  performance, 
therefore  12  subjects  were  used.  Given  this  number  of  participants,  the  study  should  be 
considered  to  be  of  relatively  high  strength  in  identifying  usability  concerns  for  the  devices 
tested. 
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Figure  1,  Usability  testing  methods  (Dumas  and  Redish,  1993) 
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2  Method 


2.1  Participants  and  Facility 

Twelve  U.S.  Air  Force  personnel  at  Hurlburt  Field  served  as  participants.  These 
individuals  were  assigned  to  the  16th  Aircraft  Maintenance  Squadron  (AMXS).  All  participants 
were  Communication/Navigation  specialists.  Testing  was  conducted  at  the  16th  AMXS,  Hurlburt 
Field,  FL. 

2.2  Time  and  Schedule 

The  testing  was  conducted  from  4  November  through  7  November  2002.  Pre-testing 
occurred  on  the  first  day  and  consisted  of  assuring  that  appropriate  software  was  loaded  on  the 
hardware  devices,  and  re-evaluating  all  scenarios  to  verify  that  the  same  types  of  manipulations 
were  required  across  devices.  Subjects  were  scheduled  on  a  non-interference  basis;  that  is, 
scheduling  was  arranged  based  solely  on  availability,  so  as  not  to  interfere  with  subjects’  regular 
work.  Testing  on  all  four  days  lasted  for  most  of  the  day  (morning  and  afternoon). 

2.3  Test  Equipment  Requirements 

Hardware  used  in  the  test  included  the  items  listed  below. 

1 .  Intermec  710  handheld  device  with  POMX  software 

2.  Extra  batteries  for  Intermec  710 

3 .  Battery  charger  for  Intermec  7 1 0 

4.  Standard  CAMS  workstation  configured  with  CAMS  “green  screens” 

2.4  Data  Collection  Packet 

Data  collection  packets  included  a  variety  of  forms.  Each  subject  was  given  a  complete 
packet  at  the  beginning  of  the  test  and  completed  all  forms  prior  to  the  end  of  the  test. 

1 .  Checklist 

2.  In  briefing 

3.  Consent  form 

4.  Pre-test  questionnaires 

5.  Post-condition  questionnaires 

6.  Post-test  questionnaires 

2.5  Data  Collection  Team 

The  data  collection  team  consisted  of  five  individuals:  1)  two  experimenters,  2)  a  subject 
matter  expert,  3)  a  videographer,  and  4)  a  study  coordinator.  The  experimenters  conducted  the  in 
briefing  and  out  briefing  sessions  and  administered  the  questionnaires.  The  subject  matter  expert 
provided  the  majority  of  the  interaction  with  test  participants.  The  videographer  operated  a 
video  camera  in  order  to  record  participants’  actions.  The  study  coordinator  was  responsible  for 
assuring  that  scheduling  was  completed  for  all  participants,  that  all  hardware  and  software  were 
available  for  the  test,  and  that  in-briefing  and  out-briefing  sessions  were  conducted  for  each 
participant. 
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2.6  Pre-Test  Requirements 

Several  activities  needed  to  be  completed  prior  to  beginning  this  test.  Some  of  these 
activities  were  accomplished  prior  to  arrival  at  Hurlburt;  others  were  completed  during  the  pre¬ 
test  (the  first  day  of  testing).  Most  of  these  activities  required  coordination  with  the  hosting 
facility.  c 

1 .  Experimenters  required  time  to  view  the  POMX  software.  This  software  was  loaded  on 
the  handheld  devices  several  months  prior  to  this  test  and  experimenters  reviewed  the 
software  with  system  developers  several  weeks  prior  to  the  test. 

2.  Experimenters  required  time  to  review  the  features  of  the  Intermec  71 0  so  that  they 
would  be  familiar  with  the  device  prior  to  the  test.  This  was  accomplished  during  the 
pre-testing  day. 

3.  Experimenters  created  and  finalized  scenarios  for  the  subjects  to  use  during  the  test. 
Scenarios  were  developed  several  weeks  prior  to  die  test  and  finalized  on  the  pre-testing 
day. 

4.  Experimenters  were  granted  clearance  to  take  photos  at  Hurlburt  Field.  Videotaped  data 
and  digital  photos  were  captured  during  the  test. 

5.  Experimenters  required  use  of  an  aircraft  parking  spot  on  the  flightline  for  testing. 

2.7  Test  Procedure 

Test  participants  were  provided  an  in  briefing  in  a  conference  room  setting.  This  in¬ 
briefing  provided  an  overview  of  the  purpose  of  the  test.  At  this  time  they  also  completed  the 
consent  form,  and  completed  the  pre-test  questionnaire.  The  study’s  original  design  included 
counterbalancing  of  subjects  across  conditions  to  address  the  possibility  of  order  effects. 
However,  due  to  early  trouble  initializing  the  POMX  database,  counterbalancing  was  abandoned. 
Instead,  each  participant  started  with  the  baseline  condition  (no  POMX).  Participants  began  the 
baseline  condition  at  an  area  just  outside  the  16th  AMXS  facilities.  While  at  this  area, 
participants  simulated  entering  maintenance  data  relevant  to  an  aircraft  inspection.  Once  the 
simulated  data  entry  task  was  completed  participants  walked  into  the  16th  AMXS  facilities  and 
used  a  CAMS  terminal  to  complete  the  appropriate  documentation  for  the  simulated  inspection. 
After  the  participants  performed  and  documented  three  simulated  data  entry  tasks  (i.e.,  open  job, 
order  parts,  and  close  job),  they  were  asked  to  fill  out  a  post-condition  questionnaire.  Time 
measurements  were  recorded  for  each  of  the  three  times  each  participant  took  part  in  the  baseline 
condition.  Time  measurements  were  also  recorded  to  determine  each  subject’s  average  travel 
time  to  the  nearest  aircraft  parking  space.  These  time  measurements  documented  both  actual 
maintenance  data  entry  task  time  and  travel  time  required  as  part  of  the  trial. 

Following  the  completion  of  the  baseline  condition,  each  subject  used  the  POMX  device 
in  a  batch  condition  in  order  to  open  a  job.  Prior  to  beginning  this  batch  condition,  each 
participant  received  training  on  the  POMX  device.  Once  trained  on  the  device,  participants 
initiated  the  simulated  inspection  task  and  the  subsequent  data  entry  task  (i.e.,  opening  a  job) 
from  the  same  area  in  which  they  had  begun  the  baseline  condition.  From  this  starting  point, 
participants  walked  to  the  aircraft  parking  spot  that  was  closest  to  the  16th  AMXS  building. 

Once  the  participants  arrived  at  the  aircraft  parking  spot  they  simulated  the  inspection  of  an 
aircraft.  Upon  completion  of  the  simulated  inspection  participants  used  the  POMX  device  to 
open  a  job  for  the  inspection.  Once  the  job  was  successfully  opened  participants  walked  back  to 
the  starting  area.  Following  this  process  participants  were  asked  to  fill  out  a  post-condition 
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questionnaire  and  a  post-test  questionnaire.  Time  measurements  were  recorded  for  the  POMX 
batch  condition.  Again,  these  time  measurements  documented  both  actual  data  entry  task  time, 
and  travel  time  required  as  part  of  the  trial. 
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3  Results 


The  data  presented  in  the  results  section  of  this  report  compares  the  POMX  condition  and 
the  baseline  CAMS  condition  for  only  one  of  the  three  critical  functions  of  the  POMX  system, 
that  of  opening  a  work  order.  The  other  two  critical  POMX  functions,  ordering  parts  and 
closing  work  orders  were  not  operating  sufficiently  to  be  evaluated  at  the  time  this  testing  was 
conducted.  Therefore,  it  is  important  to  note  that  these  data  do  not  reflect  an  overall  evaluation 
of  the  POMX  system  or  its  acceptability  and  usefulness  in  its  current  state.  Instead,  these  data 
show  the  comparison  of  one  functioning  aspect  of  the  system  with  the  currently  utilized  CAMS 
terminal  system.  For  a  summary  of  additional  key  findings,  observations,  and  recommendations 
regarding  the  entire  POMX  system  at  the  time  of  testing  please  see  the  Summary  and  Key 
Findings  section  at  the  end  of  this  report. 

Analysis  of  the  data  collected  during  this  study  included  direct  comparison  of  actual  trial 
time  required  to  complete  specific  maintenance  activities,  as  well  as  subjective  evaluations  of 
both  the  POMX  system  and  the  CAMS  system.  These  analyses  are  detailed  in  the  following 
sections.  It  is  important  to  note  that  a  triangulation  method  was  used  in  the  analysis  of  findings 
for  each  device.  Triangulation  was  implemented  as  defined  in  the  following  paragraphs. 

Participants  completed  various  trials  (e.g.,  the  open  job,  order  parts,  and  close  job  tasks) 
associated  with  aircraft  maintenance  in  two  different  conditions:  the  CAMS  open  work  order, 
order  part  and  close  work  order  conditions  (which  simulated  their  typical  task  experience)  and 
the  POMX  open  work  order  condition  (which  utilized  a  handheld  device  to  allow  remote 
completion  of  the  above  maintenance  tasks).  The  completion  times  for  these  trials  are  discussed 
in  the  analysis. 

A  comparison  was  made  between  the  relevant  completion  times  for  the  various 
maintenance  activities.  While  three  maintenance  tasks  were  completed  in  the  CAMS  condition, 
only  the  ‘open  job5  task  was  completed  in  the  POMX  condition.  Thus,  the  trials  times  associated 
with  opening  a  job  were  compared  across  the  two  conditions. 

Ratings  for  both  devices  were  gathered  for  each  item  in  the  questionnaire  (e.g,,  for  each 
question),  and  mean  ratings  and  standard  deviations  were  then  calculated  on  each  item  for  that 
device.  Ratings  could  range  from  1  to  5  (a  rating  of  1  was  considered  a  positive  response  and  a 
rating  of  5  was  a  negative  response). 

A  priori,  criteria  were  set  that  1)  any  item  with  an  average  rating  of  >  3  indicated  a 
usability  problem,  and  2)  any  item  where  the  total  of  the  average  rating  plus  standard  deviation 
was  >  3  indicated  a  potential  usability  problem. 

Once  this  initial  analysis  was  complete,  user  ratings  were  plotted  on  a  cluster  graph  for 
items  that  indicated  usability  problems  or  potential  usability  problems.  User  comments  and 
observer  notes  were  then  analyzed  and  collapsed  to  assist  in  the  definition  and  clarification  of  the 
problem  or  potential  problem.  That  is,  user  comments  and  observer  notes  that  related 
specifically  to  the  item  were  added  to  the  analysis  to  assist  in  further  definition  of  the  problem  or 
potential  problem. 

3.1  Objective  Results 

In  this  study,  participants  completed  maintenance  data  entry  tasks  in  a  baseline  condition 
-  a  condition  that  replicated  the  manner  in  which  the  aircraft  maintenance  data  collection  is 
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•currently  completed  -  and  then  completed  one  identical  data  entry  task  (i.e.,  open  job)  in  the 
POMX  condition.  Start  and  stop  times  were  recorded  for  each  of  these  trials.  The  time  required 
to  complete  the  open  job  task  in  each  condition  was  then  compared  for  each  subject. 

Statistical  analysis  of  this  time  data  shows  that  the  POMX  condition  shows  significant 
time  improvements  over  the  CAMS  condition.  This  was  explored  using  a  dependant  samples  t- 
test  conducted  on  the  total  trial  time  for  the  open  job  task  in  the  baseline  and  POMX  conditions. 

For  the  first  comparison,  only  the  actual  times  to  complete  the  trial  were  compared.  This 
excluded  the  time  required  for  the  maintainer  to  travel  from  the  aircraft  to  the  CAMS  terminal, 
and  the  time  required  to  return  to  the  aircraft  after  receiving  a  response  from  the  CAMS  system. 
As  hypothesized,  maintainers  took  more  time  to  complete  the  open  job  task  in  the  CAMS 
condition  than  in  the  POMX  condition  [t(l  1)  =  2.1 15,  p<.05].  In  the  POMX  condition, 
participants  spent  an  average  of  0:05:41  completing  the  open  job  task,  while  in  the  CAMS 
condition  participants  spent  an  average  of  0:08: 1 3  completing  the  open  job  task. 

To  include  travel  times  in  the  calculation,  experimenters  recorded  the  time  required  for 
each  maintainer  to  travel  from  the  CAMS  terminals  to  the  nearest  aircraft  parking  location 
(approximately  300  feet).  This  time  was  recorded  at  least  twice  for  every  participant  -  three 
times  for  most  -  and  these  times  were  averaged  to  determine  a  subject’s  travel  time  to  include  for 
calculations  in  this  study.  It  should  be  noted  that  these  were  travel  times  to  the  nearest  aircraft 
parking  location.  A  small  sample  of  travel  times  to  the  most  distant  aircraft  parking  location  was 
also  gathered  for  informational  use,  but  was  not  used  in  comparing  the  baseline  and  POMX 
conditions. 

As  hypothesized,  the  time  required  to  complete  the  trial  in  the  POMX  condition  was 
significantly  less  than  the  time  required  in  the  CAMS  condition  when  time  to  complete  the  entire 
trial  was  considered  [t(  11)  =  4.173,  p<.01].  The  POMX  condition  showed  the  same  average  of 
0:05:41  to  complete  the  open  job  task.  The  CAMS  condition  showed  an  average  of  0:10:46  to 
complete  the  open  job  task  when  travel  to  and  from  the  aircraft  to  interact  with  the  CAMS 
system  was  included  in  the  calculation.  Figure  2  shows  a  comparison  of  average  POMX  and 
CAMS  trial  times,  both  with  and  without  travel  times. 
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Figure  2.  Average  Trial  Times  for  the  Baseline  and  POMX  Conditions 

3.2  Subjective  Resuite 

Participants  completed  maintenance  data  collection  activities  (i.e.,  open  job,  order  parts, 
close  job)  as  an  initial  baseline.  During  these  trials,  subjects  maintained  the  normal  sequence  for 
activities  associated  with  performing  the  inspection  of  the  aircraft.  Participants  later  completed 
the  “open  job”  maintenance  data  collection  activity  in  the  POMX  condition. 

After  completing  the  maintenance  tasks  in  each  condition,  participants  rated  each  device 
across  a  series  of  elements  using  a  5-point  scale,  where  one  was  positive  and  five  was  negative. 
Participants  rated  the  acceptability  of  each  device  for  such  characteristics  as: 

•  display  size, 

•  computer  size, 

•  pointing  device, 

•  keyboard  layout, 

•  ease  of  navigating, 

•  ease  of  using  drop-down  menus,  and 

•  frustration  level  when  using  the  device. 

Additionally,  participants  rated  the  Intermec  710  (the  POMX  device)  across  another  four 
elements  specific  to  the  use  of  a  handheld  device  when  entering  data  on  the  flightline;  these 
additional  elements  included  ruggedization,  ease  of  using  the  onscreen  keyboard,  reading  data, 
and  perceived  portability. 
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In  all  categories,  average  ratings  for  the  POMX  condition  were  in  the  acceptable  range; 
that  is,  the  average  rating  in  each  category  was  not  greater  than  or  equal  to  three.  Based  on  this 
rating,  it  is  concluded  that  there  were  no  demonstrated  usability  problems  in  the  POMX 
condition.  Furthermore,  the  average  rating  plus  the  standard  deviation  remained  within  this 
acceptable  range  across  all  categories  in  the  POMX  condition;  this  indicates  that  no  potential 
usability  problems  were  revealed  in  the  POMX  condition  in  this  study  (Figure  3,  Figure  4). 
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Figure  3.  Means  and  Deviations  -  Acceptability  Ratings  for  the  POMX  Condition 
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Figure  4,  Means  and  Deviations  -  Frustration  Ratings  for  the  POMX  Condition 


In  the  baseline/C  AMS  condition,  the  average  ratings  in  each  category  were  in  the 
acceptable  range,  indicating  no  demonstrated  usability  problems  with  any  of  the  items  rated  in 
the  baseline  condition.  However,  the  calculation  of  the  average  rating  plus  standard  deviation 
revealed  two  potential  problems  in  the  baseline  condition  (Figure  5,  Figure  6), 
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Figure  5.  Means  and  Deviations  -  Acceptability  Ratings  for  the  Baseline  Condition 
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Figure  6.  Means  and  Deviations  -  Frustration  Ratings  for  the  Baseline  Condition 
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Both  the  Ease  of  Navigation  and  Frustration  Level  categories  showed  ratings  whose 
deviation  extended  into  the  unacceptable  range.  Individual  ratings  for  these  usability  categories 
are  illustrated  in  Figures  7  and  8. 

Individual  ratings  of  navigation  in  the  baseline  condition  indicated  a  potential  problem. 
While  ten  participants  provided  ratings  in  the  acceptable  range,  one  participant  rated  navigation 
using  paper  and  the  CAMS  system  as  Moderately  Unacceptable,  and  another  participant  rated 
this  as  Extremely  Unacceptable  (Figure  7).  Experimenters  noted  that  participants  had  some 
difficulty  navigating  within  CAMS  for  various  reasons.  Some  participants  experienced  problems 
logging  into  CAMS:  several  participants  either  lacked  a  valid  login  and  password,  or  received  an 
error  when  they  input  their  existing  login  and  password.  Also,  while  some  subjects  in  this  study 
were  highly  experienced  using  CAMS,  other  participants  had  a  lack  of  experience  -  and  thus 
familiarity  -  with  CAMS. 
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Figure  7.  Individual  Ratings  for  Navigating  in  the  POMX  and  Baseline  Conditions 


The  indication  of  a  potential  problem  with  the  frustration  level  in  the  baseline  condition 
may  be  partially  influenced  by  the  difficulty  participants  expressed  with  navigating  the  CAMS 
system.  Only  five  participants  provided  ratings  in  the  acceptable  range  when  asked  about  their 
frustration  level  in  the  baseline  condition.  Five  participants  rated  their  frustration  levels  as 
Neutral,  and  two  additional  participants  rated  their  frustration  levels  as  Extremely  High  (Figure 
8).  In  addition  to  the  difficulties  with  CAMS  mentioned  above,  it  was  noted  that  some 
participants  had  difficulty  finding  an  available,  functioning  CAMS  terminal,  or  the  system  may 
have  frozen  while  they  were  using  it.  In  some  cases,  the  CAMS  system  rejected  information  that 
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had  been  entered,  and  participants  were  required  to  complete  the  process  again.  Experimenters 
noted  that,  while  participants  did  experience  some  delay  and  difficulty  navigating  when  using  the 
CAMS  system,  this  reported  frustration  level  could  easily  reflect  users’  long-term  experiences 
with  the  system,  and  not  simply  their  experiences  during  this  test.  For  example,  these  ratings 
could  reflect,  in  part,  the  long  data  entry  sessions  often  associated  with  users’  experience  with 
CAMS. 
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Figure  8.  Individual  Ratings  for  Frustration  in  the  POMX  and  Baseline  Conditions 


4  Discussion 


The  purpose  of  this  study  was  to  provide  subjective  and  objective  data,  based  on  standard 
usability  testing  methods,  comparing  the  usability  of  the  POMX  system  and  device  for  opening 
work  orders  in  a  batch  mode  to  the  current  maintenance  methodology  that  utilizes  paper  and  the 
CAMS  system  for  opening  jobs. 

From  the  standpoint  of  objective  data,  it  was  hypothesized  that  the  POMX  condition 
would  show  significant  improvements  in  the  time  required  to  complete  maintenance  at  the 
aircraft  -  that  is,  at  the  point  of  maintenance.  The  current  method  for  completing  this 
maintenance  requires  that  the  maintainer,  upon  finding  a  problem,  complete  paper 
documentation  at  the  aircraft,  then  proceed  indoors  in  order  to  use  the  CAMS  system,  the 
maintainer  signs  in  to  the  CAMS  terminal  (this  particular  step  has  been  shown  to  be  somewhat 
annoying  to  the  user),  opens  the  maintenance  job,  fills  out  the  necessary  fields  within  the  system 
and  sends  the  job,  and  then  waits  for  a  response  from  the  CAMS  system  with  a  Job  Control 
Number  (JCN).  At  this  point,  the  maintainer  signs  out  of  the  CAMS  system,  returns  to  the 
aircraft,  and  documents  the  JCN  in  the  aircraft’s  paper  maintenance  forms.  In  the  POMX  batch 
condition,  maintainers  can  remotely  access  information  from  the  CAMS  database  using  an 
electronic  handheld  device.  Thus,  the  maintainer  completes  the  paper  documentation  at  the 
aircraft,  and  fills  out  the  necessary  screens,  upon  returning  to  the  shop  the  job  can  be  downloaded 
to  the  POMX  database  and  subsequently  to  CAMS  automatically. 

.  It  should  be  noted  that  the  CAMS  condition  in  this  study  was  configured  to  represent  the 
best  possible  scenario  for  quick  entry  of  maintenance  data  collection  utilizing  the  CAMS  system. 
The  simulated  maintenance  tasks  occurred  at  the  closest  possible  aircraft  parking  location,  the 
maintainer  went  directly  from  the  aircraft  into  the  building  where  he  updated  the  CAMS  terminal 
and  received  the  response  (i.e.,  the  JCN),  and  then  the  maintainer  proceeded  directly  back  to  the 
aircraft  (in  the  nearest  aircraft  parking  space)  to  update  the  aircraft  documentation.  In  ordinary 
maintenance  activities,  the  aircraft  requiring  maintenance  is  often  not  in  the  nearest  aircraft 
parking  location,  and  the  maintainer  often  does  not  enter  data  into  CAMS  until  the  end  of  his 
shift.  Thus,  maintenance  data  in  CAMS  may  not  actually  be  updated  for  up  to  eight  hours,  and 
the  maintenance  information  at  the  aircraft  will  not  be  current  until  after  the  maintainer  has 
received  a  response  from  the  CAMS  system,  and  then  updated  the  documentation  at  the  aircraft. 

Even  so,  this  study  did  show  significant  time  savings  in  the  POMX  condition  as 
compared  with  the  CAMS  condition.  It  is  interesting  to  note  that,  even  when  the  necessary  travel 
time  required  in  the  CAMS  condition  was  excluded,  the  POMX  condition  required  less  time  than 
did  the  CAMS  condition.  This  supports  the  subjective  feedback  recorded  through  user  ratings 
that  POMX  was  more  usable  than  CAMS  for  this  particular  maintenance  data  collection  activity. 
Whereas  the  POMX  condition  was  rated  in  the  acceptable  range  in  all  usability  categories,  the 
CAMS  condition  exhibited  potential  problems  with  navigation  and  frustration  level.  Based  on 
other  user  feedback  and  experimenter  observations,  it  is  likely  that  the  sometimes-lengthy 
response  time  of  the  CAMS  system  lends  to  frustration  with  the  system.  Still,  other  issues  -  such 
as  login  difficulties,  and  difficulty  navigating  often  confusing  screens  to  access  necessary  data  - 
are  likely  to  be  equally  responsible  for  the  reported  frustration  with  the  , CAMS  system. 
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5  Summary  and  Key  Findings 


The  comparative  measurements  taken  during  this  study  indicate  that  there  are  potentially 
tremendous  gains  to  be  made  in  the  efficiency  of  maintenance  task  documentation  and  the 
associated  documentation.  For  this  study,  the  time  savings  found  when  travel  time  was 
incorporated  into  CAMS  entry  trial  times  were,  as  anticipated,  significant.  Additionally,  even 
when  the  necessary  travel  time  was  excluded,  the  POMX  condition  required  less  time  than  did 
the  CAMS  condition.  Average  travel  time  was  determined  based  on  the  location  of  the  nearest 
aircraft  parking  location.  Two  additional  time  samples  were  recorded  to  document  travel  time  to 
the  farthest  aircraft  on  the  1 5th  AMU  flightline  at  Hurlburt.  Given  that  travel  time  will  vary 
widely  based  upon  the  flightline  size  of  various  installations,  it  should  be  noted  that  flightline 
size  should  clearly  be  considered  when  estimating  the  potential  time  savings  a  unit  might  gain  by 
incorporating  devices  that  might  be  utilized  to  interact  with  necessary  maintenance  systems  from 
the  point  of  maintenance. 

As  noted  in  the  Introduction,  a  complete  test  of  the  POMX  system  was  not  possible 
during  this  testing  period.  Therefore,  the  test  was  re-scoped  to  include  only  a  limited  number  of 
conditions.  This  report  has  focused  on  providing  results  of  that  limited  test.  It  is  important, 
however,  to  report  overall  key  findings  as  they  relate  to  the  total  POMX  capability.  The  intent  of 
reporting  these  findings  is  to  provide  feedback  to  POMX  designers  for  design  modifications  that 
will  address  these  issues.  These  overall  findings  are  identified  in  the  paragraphs  below. 

A  key  finding  revealed  the  first  day  was  the  fact  that  when  jobs  are  entered  in  CAMS 
through  a  regular  CAMS  terminal,  those  same  jobs  do  not  show  up  and  cannot  be  accessed  in  the 
POMX  system.  This  may  be  due  to  the  fact  that  CAMS  and  POMX  do  not  synchronize 
frequently  enough.  Even  if  the  POMX  software  and  the  RF  systems  were  all  functioning  at 
optimal  rates,  this  would  be  a  serious  condition  that  must  be  addressed.  When  personnel  use  the 
POMX  device  they  should  be  able  to  access  any  and  all  jobs  that  are  currently  open  in  CAMS. 

Another  key  observation  was  that  during  the  parts  inquiry  with  the  handheld  device,  the 
drop  down  box  with  QRLs  could  not  be  scrolled  left  or  right  and  therefore  the  nomenclature  of 
the  aircraft  parts  that  corresponded  with  the  visible  QRL  number,  were  not  visible.  Without  the 
nomenclature  of  the  part,  the  QRL  is  of  no  value  to  the  user.  When  the  parts  NSN  were  entered 
(defeating  the  purpose  of  the  QRL),  the  system  returned  the  QRL  numbers. 

To  summarize  the  overall  findings  of  this  testing,  the  observations  produced  good  data 
for  the  baseline  CAMS  condition  and  identified  numerous  software  problems  with  the  year-one 
effort.  The  following  list  indicates  the  observations  of  this  test  in  the  order  of  importance  based 
on  the  opinion  of  the  testing  team. 

] .  CAMS-POMX  data  synchronization  does  not  occur  often  enough  (should  be  real  time). 

2.  Wait  time  for  drop  down  menus  (for  instance  accessing  work  center  data  when  ordering 
parts)  is  unacceptably  long  (up  to  20  minutes). 

3.  Variation  of  signal  strength  is  affecting  system  performance. 

4.  Server  availability  (functionality)  is  intermittent  and  not  reliable  enough  for  continued  daily 
use. 

5.  Daily  database  table  updates  are  not  reliable  (usually  only  2  or  3  devices  out  of  1 6  would 
update  properly  or  “refresh”  each  morning). 
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These  key  areas  must  be  improved  before  a  complete  objective  test  of  the  POMX  system 
can  be  accomplished.  After  the  identified  problems  are  corrected  and  a  fully  functional  POMX 
system  is  available,  it  is  recommended  that  the  AFRL  team  return  to  Hurlburt  Field  and  re-take 
the  time  measurements  for  the  POMX  system  based  on  direction  from  HQ  AF/ILMM. 

This  study  lays  excellent  groundwork  for  future  studies  addressing  the  usability  and 
efficiency  of  current  and  proposed  POMX  maintenance  information  systems.  All  baseline  data 
for  the  open  job,  order  parts,  and  close  job  tasks  was  collected  for  the  CAMS  condition,  and  each 
of  these  areas  are  valid  areas  for  comparison  with  alternate  maintenance  systems.  It  is 
recommended  that  the  study  be  conducted  again  when  the  functionality  of  the  POMX  database  is 
such  that  other  tasks  might  be  executed  and  compared  with  the  existing  system.  That  is,  after 
Year  2  software  is  installed  it  is  recommended  that  further  testing  of  the  POMX  system  be 
conducted.  Specifically  the  POMX  live  open  work  order,  order  parts,  and  close  work  order 
conditions  should  be  tested  as  well  as  the  POMX  batch  order  parts  and  close  work  order 
conditions.  Only  this  thorough  testing  approach  will  truly  reveal  potential  savings  offered  by  a 
POMX  capability. 
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